Object extraction techniques and visual image search
with Semantic web techniques
Aninda Maulik
Supervisors: Prof. Pierre Maret
Dennis Diefenbach
Cyber Physical and Social Systems
University of Jean Monnet
July 2020
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 1 / 42
Introduction:Just Google
images with bicycles
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 2 / 42
images with bicycles on the right
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 3 / 42
Introduction:QAnswer
QAnswer is knowledge based QA system.
A knowledge base is a collection of facts that can be interpreted by a
machine
Such a fact can look like this:”bicycle” ”in” ”right”.
QAnswer translates from a natural language question to correct
SPARQL queries.
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 4 / 42
Content
Implementation of an Algorithm for object extraction.
Design of a semantic web modelling for extracted data.
Implementation of a visual image search engine through Qanswer.
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 5 / 42
Implementation of an Algorithm for object extraction.
YOLO-(You Only Look Once)
Implementation of an Algorithm for object extraction.
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 6 / 42
Class Number, Class Name, QID
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 7 / 42
Class Number, Class Name, QID
https://github.com/pjreddie/darknet/blob/master/data/
coco.names
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 8 / 42
Bounding Box
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 9 / 42
Co-ordinate representation of Bounding Box
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 10 / 42
Image-Object Relations
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 11 / 42
Algorithms for Image-Object Relations
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 12 / 42
An attempt to show a graphical representation of top,
bottom, left, right, and central position
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 13 / 42
Algorithms for Image-Object Relations
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 14 / 42
An attempt to show a graphical representation of
Image-Object relation in comparison to an incoming photo
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 15 / 42
An attempt to show a graphical representation of
Image-Object relation
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 16 / 42
Using Image-Object relation we create a csv file
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 17 / 42
Design of a semantic web modelling for extracted data.
Image-Object
Design of a semantic web modelling based on the csv file.
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 18 / 42
Object-Object Relation
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 19 / 42
OORelation algos ready to be used
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 20 / 42
An attempt to show a graphical representation of O-O
Relations in comparison to an incoming photo
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 21 / 42
OORelation algos ready to be used, contd..
Greater and Smaller Than
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 22 / 42
An attempt to show a graphical representation of O-O
Relations in comparison to an incoming photo
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 23 / 42
OORelation algos ready to be used, contd..
Overlapping With
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 24 / 42
Object-Object Relation
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 25 / 42
Reified Triple
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 26 / 42
Regular Triple
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 27 / 42
Knowledge Graph
Image-Object Relation with a hint of Object-Object Relation
Design of a semantic web modelling for extracted data.
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 28 / 42
After Semantic Web Modelling
1
We convert the CSV file created for Image-Object Relations into
a RDF file
with the help of a Java Program
2
and upload the RDF file into QAnswer
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 29 / 42
Results
QAnswer: images with bicycle in the right
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 30 / 42
images in the center
QAnswer: airplane in the center
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 31 / 42
images in the left
train in the left
https://qanswer-frontend.univ-st-etienne.fr/user/query?
kb=onto&user=anindamaulik
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 32 / 42
Qanswer: Image-Object Relation–contains
person and chair
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 33 / 42
special Wikimediacommons api give images and human
hand-annotated structured data
The Api
https://commons.wikimedia.org/w/api.php?action=query&list=
search&srsearch=haswbstatement:
P180=Q7378&srnamespace=6&format=json
give images in form of json data
and human hand-annotated structured data for some images
Examples
1
what are the items portrayed in the image
2
who is the creator of the file
3
what is the source of the file
4
inception date of the file
5
copyright status
6
The url showing structured data
https://commons.wikimedia.org/wiki/File:
African_elephants,_Lake_St_Lucia_06.jpg
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 34 / 42
Automation pipeline
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 35 / 42
Limitations and Future Work
Query for images on the top
Issue with confidence of detection by YOLO
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 36 / 42
YOLO got overconfident
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 37 / 42
Reified Triple
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 38 / 42
Reified Triple not being interpreted by QAnswer
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 39 / 42
special Wikimediacommons api give images and human
hand annotated structured data
Our time constraints
https://commons.wikimedia.org/w/api.php?action=query&list=
search&srsearch=haswbstatement:
P180=Q7378&srnamespace=6&format=json
We are yet to handle the incoming RDF data from the api
and merge with our RDF file,
in order to be able to query
”Give me pictures of bicycles from February 2019”
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 40 / 42
Conclusion
1
We have worked on improving image search engines
2
Computer Vision techniques is able to identify objects in images.
3
Semantic Web techniques give a semantic representation of the
images that can be queried QAnswer.
4
This work can be easily used by any search or query engine to give
results
5
We do not have a bench mark as of yet
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 41 / 42
END
Thank you for your attention, time and patience
Please ask me any question that you have.
Please provide your suggestion which can be used to make my work
better
Aninda Maulik (University Jean Monnet) Object extraction techniques and visual image search with Semantic web techniquesJuly 2020 42 / 42